Distributed Algorithms for the Transitive Closure

Many database queries, such as reachability and regular path queries, can be reduced to finding the transitive closure of the underlying graph. For calculating the transitive closure of large graphs, a distributed computation framework is required to handle the large data volume (which can approach O(|V |) space). Map Reduce was not originally designed for recursive computations, but recent work has sought to expand its capabilities in this regard. Distributed, recursive evaluation of the transitive closure faces two main challenges in the Map Reduce environment: a large number of rounds may be required to find all tuples in the transitive closure on large graphs, and the amount of duplicate tuples derived may incur a data volume cost far greater than the size of the transitive closure itself. Seminaive and smart are two algorithms for computing the transitive closure which make different choices in handling the tradeoffs between these two problems. Recent work suggests that smart may be superior to seminaive in the Map Reduce paradigm, based on an analysis of the algorithms but without an accompanying implementation. This view diverges from earlier work which found seminaive to be superior in the majority of cases, albeit in a non-distributed environment. This paper presents implementations of seminaive and smart transitive closure built upon the Hadoop framework for distributed computation and compares the performance of the algorithms across a broad class of data sets, including trees, acyclic, and cyclic graphs.